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INTRODUCTION 



; Open learning courses raise a number of research and evaluation 
tissues. This is partly explained by the much more comprehensive in- 
^^structional design characteristics of open learning courses compared 
:to conventional classroom instruction. One question which has been 
^pursued at Chicago's TV College over the years has been an accurate 
^prediction of completion rates— both overall and individual. 

In 1973, a cooperative arrangement between the City Colleges of 
; Chicago and the University of Chicago's Measurement, Evaluation, and 
Statistical Analysis (MESA) Program brought two part time research interns 
to work on a variety of evaluation issues at TV College. The need to 
tag students according to their potential for completion of the TV 
College courses was given high priority. 

An instrument was developed which attempted to assess such things 
as attitudes toward television-based instruction, studying at home, and 
some thirty other factors which were felt relevant for separating those 
vyho might drop out from those who would not. Results using the attitude 
instrument were inconclusive despite comprehensive efforts by the first 
and second generation MESA interns. One nearly overlooked side effect 
was singularly outstanding and in keeping with experiences of others 
conducting mail -in questionnaire studies. Those who returned the 
questionnaire were distinctly different from those who didn't on one 
important criterion: they tended to complete the course eight times 
out of ten.- . 

The emphasis on the pilot study had been on identifying individuals 
at registration time who might require extra attention in order to complete 
the course. This still remains a long term objective. However, the 
third generation MESA research intern (senior author) has picked up 
the task in a different manner and has concentrated on prediction of - 
overall completion rates. After identifying key factors for predicting 
overall rates, we hope to work backward and establish non-obtrusive 
measures for estimating probabilities for individual completions. This 
study reports on our progress in predicting overall completion rates 
for various open learning courses given the demographic, curricular, 
and administrative conditions prevailing at the City Colleges of Chicago. 

BACKGROUND AND DEFINITIONS 

Chicago's TV College is now beginning its twentieth year of pro- 
viding open broadcast credit courses. It is part of a larger open learning 
system which includes course offerings by radio, newspaper, and video- 
cassette in addition to open broadcast television. These courses use 
materials produced locally, leased from others, and recent PBS series 
adapted for college credit (Ascent of Man, Classic Theater, and Adams' 
Chronicles ) . . 
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The student characteristics for the television courses are typical 
of open learning experiences elsewhere: 

—60% female 
--average age 32 years 
--working at least part time 

--50% are enrolled on campus as well as for television 
--50% are caking a television course for the first time 
—most take television courses out of necessity prompted by working, 
children at home, convenience, etc. 

For purposes of this study, several definitions need to be kept 
in mind. Course completion is defined as achieving an A,B,C, or D 
grade at the conclusion of the 15 week semester. Incompletes (R) are 
treated as a withdrawal (W) to simplify our record keeping since students 
nave two , semesters to remove the R grade. There Is typically a 10% 
incomplete rate with many Incompletes reverting to W or F. Failure 
{f) is considered a withdrawal or unsuccessful completion of the course 
since the student receives no credit for an F grade. 

The base enrollment figure for a course is the initial registration 
number rather than a mid-term or ten day figure. Thus we are treating 
gross enrollment figures rather than adjusted figures. This tends to 
inflate the withdrawal rate but Is the only feasible measure since 
we cannot conduct a head count, as is possible in the classroom 
cnc, criteria, a typical TV College course would have a 

60% withdrawal figure. Previous TV College reports (Zigerell and 
Chausow, 1974) show lovyer withdrawal rates. While there has been a slow 
increase in withdrawal, figures, the difference Is partly explained by 
previous use of a two tier withdrawal accounting procedure. Official 
withdrawal occurs when the student files for withdrawal status. Unofficial 
withdrawal is recorded when the student does nothing after Initial 
registration. We do not make this distinction here In order to simplify 
our accounting method and to more accurately describe withdrawal rates 
Our approach also allows greater comparison across -systems using different 
methods of defining withdrawals. 

It has been a long standing practice for TV . lege students to 
complete an evaluation instrument early in the ."I '- ter, as one of 
their course requirements. The one page question. . re is given out 
at registration and returned at the end of the tni -d week of the semester 
This in-process procedure represents a type of fomwtlve evaluation 
IScriven, 1967) where the results can be used to modify and Improve a 
course while It Is still in progress. This view is similar to those 
found at the University of Mid-America (see e.g. Gooler, 1975) and 
Coast CoiTinunity College (see e.g. Ruth and Teraz, 1974). Unit evaluation 
forms, student diaries and interviews are respectively used to generate 
information to Improve televised offerings. 

^^^fco^i^se systems depend upon only an end-of-course student 
evaluation instrument (see e.g. Telecourse Evaluation . 1975). This 
^P^oach has been labeled as surmiatlve evaluation (Scrlven 
1957) and involves judging the merits of a completed Instructional 
sequence. While there are strengths and weaknesses In both of these 
types of evaluation, the formative approach encourages the researcher 
to examine data at an early stage of the course, which allows 
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modifications in subsequent units. 
DEVELOPMENT OF THE PREDICTIVE FORMULA 



With this general orientation in mind, the four TV College courses 
offered in the Spring 1975 semester (Child Development 101, Business 
101, Economics 201, and Mathematics 111) were examined to determine if 
certain repetitions or patterns would surface. The analyses of the 
data from these courses have clearly illustrated that trends and patterns 
do exist and can serve as baseline information for later courses. 
Particularly noteworthy are the following three main factors (patterns): 

(1) sex effect ■ female open learning students complete their 
courses more' often than do their male counterparts, 

'.2) campus effect - open learning students who register at certain 
City College campuses consistently complete their courses 
more often than do students who register at other campuses, 
and 

(3) questionnaire effect - students who complete and return an 
assigned student questionnaire are much more likely to 
complete their courses than are students who do not complete 
this assignment. 

Two further secondary relationships have been found which involve 
the interaction of the sex and questionnaire variables: 

(1) females who complete the questionnaire are more likely to 
complete their open learning course than are males who 
complete the questionnaire, and 

(2) females who do not complete this instrument are more likely 
to complete their course than are males who do not submit 
the student questionnaire. 

Using results from the baseline courses, a numerical value has 
been attached to each dimension of the factors described above. For 
example, 45% of all females and 56% of all males^ completing the question- 
naire withdrew from their open learning course^.' Therefore, "weightings" 
or coefficients of .45 and .56 were assigned to female and male question- 
naire groups respectively. Each factor was weighted according to the 
results from the combined baseline courses. These weighted variables 
were than used in formulas which, it was hoped, could accurately pre- 
diet the total number of student withdrawals in subsequent open learning 
courses. 

Two separate rormulas were tested with a Music 121 course offered 
over radio. The first equation utilized u sex-by-questionnaire. inter- 
action formula. The second used the differential completion rates of 
students registered dtthe seven City College campuses. 
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The formulas are: 

(1) Sex-by- QuestlonnaIre Interaction 

W = .85(NQ) + :56(MQ) + .45(FQ) 

Where: W = the number of withdrawals 

NQ = the number of individuals who did not complete the student" 
questionnaire 

MQ = the number of male students who completed the questionnaire 

FQ = the number of female students who completed the questionnaire 

(Numerical estimates are based on 1975 rates for withdrawals.) 

(2) College of Registration 

W= .791(KK) + .659(LP) + .949(MX) + .482(MF) +.835(0H) +.579(SW) + .544(WR) 

Where: W = the number of withdrawals 

KK = the nurnber of students registered at the Kennedy-King campus 

LP = the number of students registered at Loop campus 

MX = the number of students registered at Malcolm X campus 

MF = the number of students registered at Mayfair campus 

OH = the number of students registered at Olive-Harvey. campus 

SW = the number of students registered at Southwest campus 

WR = the number of students registered at Wright campus 

(Numerical estimates are based on 1975 rates for withdrawals. Once 
again, each coefficient represents the percent.of withdrawals for a 
given variable, i.e. sex, particular campus, those submitting 
questionnaires, etc.) 

As the first trial, the number of students registered at each of 
the campuses for the Music 121 course was substituted into the campus- ' 
based formula. Likewise, the appropriate numbers of students were 
substituted into the sex-by-questionnaire formula. The results very 
closely fit the actual data. Based on the Official Class List submitted 
by the instructor at the end of the course, 59 out of 98 students did 
not complete Music 121. The sex-by-questionnaire formula predicted 
66 withdrawals. The campus-based index estimated that 65 students" 
would not complete the course. Thus, the estimates were 95.7% and 
94.2% of the actual number of withdrawals for this course. These 
results were extremely encouraging and, as a result, a number of further 
trials have been undertaken. 

The withdrawal equation has undergone a number of refinements 
which have continued to sharpen its predictive powers. The two separate 
formulas have been unified into one appropriately weighted index. 
Separate male and female estimates have been generated for the students 
not completing the questionnaire. Finally, the coefficients have been 
updated by incorporating the results from 1976 open learning course 
data. Once this method of estimation is established, it is easily 
updated by incorporating subsequent findings into the baseline data 
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pooK The combined results then become the source of new coefficients 
used in the next round of predictions ♦ This procedure is asymptotic 
in nature, in that its predictions will theoretically approach perfect 
accuracy as the number of observations increases indefinitely. 

The following data provide results of the subsequent trials of the 
prediction equation: course number, medium of presentation, enroll- 
ment, estimate of non-completions, actual number of non-completions and 
a comparison of predicted to actual. 

Business Law 211: (Locally Produced Television) - 453 registered 
students. The estimated number of non-completions was computed to 
be 286. The actual number of non-completions at the end uf the 
course was found to be 298. The predicted number was 96% of ' the 
actual withdrawal figure. 

Environmental Studies 102 : (Half Local, Half National ly-Oriented 
Television) - 596 registered students. , Estimated non-completions 
= 370. Actual number of non-completipns = 393. The predicted . 
number was 94% of the actual withdrawal figure. 

Political Science 201 : (Locally Produced Television) - 316 
registered students. Estimated non-completions = 202. Actual 
number of non-completions = 225. The predicted number was 79% 
of the actual withdrawal figure. 

Psychology 201 : (Nationally Distributed Television) - 500 registered 
students. Estimated non-completions = 343. Actual number of 
noncompletions = 339. The predicted number was 101% of the actual 
withdrawal figure. 

Social Science 106 : (Nationally Distributed Course-by-Newspaper; - 
253 registered students. Estimated number of non-completions = 
180. Actual number of non-completions = 173. The predicted number 
was 104% of the actual withdrawal figure. 

Mathematics 111 : (Locally Produced Television - eight week summer 
term) - 629 registered students. Estimated number of non-completions 
= 486. Actual number of non-completions = 537. The predicted 
number was 91% of the actual withdrawal figure. 

Further trials beyond the Mathematics 111 course (Summer semester 1976) 
have not yet been completed. 

The most recent version of the open learning withdrawal prediction 
formula is presented: 

*W = (.89(MNQ) + .71(FNQ) + .56(MQ) + .45(FQ) + .823(KK) + .668(LP) + 
.908(MX) + ,585(NE) + .871(0H) + .594(SW) + .577(WR))/2 

Where: W = the number of withdrawals 

MNQ = the number of males not submitting the questionnaire 
FNQ = the number of females not submitting the questionnaire 
MQ = the number of males submitting the student questionnaire 
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FQ = the number of females submitting the student questionnaire 

KK = the number of students registered at Kennedy-King campus 

LP = the number of students registered at Loop campus 

MX = the number of students registered at Malcolm X campus 

NE = the number of students registered at Northeast campus 

OH = the number of students registered at Olive-Harvey campus 

SW = the number of students registered at Southwest campus 

WR = the number of students registered at Wright campus 

(Based on 1975 and 1976 baseline studies.) 

*This version of the formula consists of two main factors (a campus- 
based element, and a sex-by-questionnaire component) which appear to 
have an equal bearing upon the accuracy of the final prediction. The 
division by the number 2, in effect averages the influence given to each 
of the two factors. One of the main assets of this type of formulation 
is its adaptability. Any number of additionally identified components 
can be added on, and their weight can be easily adjusted by the inclusion 
of an appropriate divisor. 



OBSERVATIONS FROM FINDINGS 

While the effectiveness of the individual predictions have varied 
slightly from course to course, they are, on the whole, quite accurate. 
Of the eight trials of the withdrawal equation (with seven different 
open learning courses), seven estimates have been within + 10%, Six 
of the eight estimates have been within +;6%. Considering the wide 
diversity in course content and delivery mode, the demonstrated 
effectiveness of the withdrawal equation is seen as very encouraging. 
A striking observation about the present findings is that course delivery 
mode appears to be quite independent of student withdrawals. The same 
predictive formula seems to apply equally well to courses presented 
via different media. It quite possible that the withdrawal variable 
is not very highly related to the delivery mode, as long as the courses 
are produced at a given level of attractiveness and are not overly 
demanding of written assignments, prerequisites, etc. These results 
lend sme credence to the view that prior educational experiences, 
administrative concerns., and other nonmedia-related components may 
be as important as a course's media component. 



USES FOR METHODOLOGY AND DATA 

The value of accurate prediction of final course completion rates 
is not simply an evaluator's exercise. Planning is underway for 
developing open learning counsel ing services. These services would actively 
and aggressively help the open learning student complete his or her 
coui^ses, or persuade students unsuited for such courses to look for 
other opportunities. Such services are currently offered under the 
Study Unlimited program which places videocassettes of TV College and 
other courses in selected Chicago Public Library branches. Counseling 
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services are expensive and requir- careful management. 

The conventional way to test a new idea is to establish experimental 
and control groups based upon random assignment. To test the open 
broadcast counseling effort in this manner would Le very difficult. 
Fairness to students, adequate control, administrative problems of" 
separating two groups, and other issues would be impossible barriers 
for conducting good research. We can compare the final course figures 
after using counseling services with predicted figures which assume no 
intervention. This simulates an experimental design. Cost/effectiveness 

and cost/benefit analyses could be determined in this manner if credence 

is given to the withdrawal formula. 

Another use of the data for decision making is the flagging of 
courses which perform above or below the expected completion rate. 
A departure from the prediction would indicate strength or weakness 
in the instructional design of individual courses. This assumes that - 
courses share common administrative procedures, scheduling, and services. 
An e.-ample of such a cour?e is Political Science 201. This course 
has a 15 percentage point difference below the average estimate. (79% 
vs. 94%). Followup studies by phone interviews have i.ot been made but 
analysis of student questionnaires suggests problems \nth the television 
presentation. The course is co.nparatively old, the instructor may have 
alienated some students, aiid the number of assignments may have been 
excessive. In any case, the weaknesses in a course's instructional 
aesign can be determir-ed by first noting a lower expected completion 
rate. This can be followed by evaluation of the course components. 

Another example of course flagging came with Mathematics 111. 
This course was offered in an eig'-t week summer session rather than the 
normal 15 week semester. We hypothesize that a "summer" effect may 
occur in accelerated courses. This causes withdrawal by some students 
who might be expected to complete an ordinary semester course. The 
"summer effect" would then skew our data toward higher than expected 
withdrawals. This explanation assumes that there was satisfactory course 
design and administration. Other evaluative procedures would confirm 
or deny this assumption. 

The practical administrative uses of the formula and the data 
generated from it are nunerous: 

--predicting the course load for instructors and thereby adjusting their 
work load accordingly; 

— the number and size of examination rooms can be planned in advance 
along with proctor requirements; 

—campuses which register TV students having low performance can be 
singled out for special attention. 



FUTURE REFINEMENTS 



We noted at the outset that our goal was to predict an individual's 
probability for completion of open learning courses. Prediction of 
overall rates is one step toward that goal. We row feel that it would 
be feasible to develop a technique for classifying students according to 
their "risk" of withdrawing. This would mean establishing a matrix of 
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relevant characteristics an.' combinations of characteristics sensitive 
J?.?""^ ^5?^"^ withdrawal. At registration time (or shortly thereafter), 
high, medium and low risk groups could be designated. Those In- the high 
risk groups could be contacted individually and receive the necessary 
counseling to determine the students' home situation, prior educational ' 
background, and interest in the course. 

m.n*. the lower risk groups might require attention and encourage- 

ment toward the end of the course and at examination times. The load 
for tne counselors could be soread out over the course by concentrating 
on the ^igh risk groups at the beginning. As the inevitable withdrawals 
take place, the.y could pick up the slack with the lower risk groups 
toward the end w^en assignments and the final examinations loom large. 
n.nw-;S^ attempting to use the tools of evaluation and research to 
provide for individual needs In what has become a very massive and 
necessarily more impersonal learning environment classroom. Personal 
attention could be focused on those students identified as requiring 
supportive services. Yet these added services require oversight by qood 
management and attention to costs. ^ijntoygooa 

We know little about the learning habits and cognitive styles 
^L'^M!? adults-partlcularly where self-study and educational media 
are involved, while we have found that the delivery mode does not seem 
to De a major factor in our predictions. It is with some unease that 
we let the issue go unexplored. Perhaps others could also pursue the 
■complex relationships which are at work with the help of the modest 
technique discussed in this report. 



SUMflARY 

This study describes a method of accurately estimating open learninn 
course completion rates based upon non-obstrusive measures made at the 
beginning of courses. The prediction formula is not dependent upon a 
particular delivery mode since the same formula was applied with equal 
precision to radio, newspaper, and open broadcast television courses 
The common threads which ran through the courses were: paced rather 
than self-paced instruction, similar administrative procedures, and 
roughly equivalent student demographic characteristics. 

The uses for an accurate overall estimate of course completion 
rates are primarily administrative. The goal of predicting Individual 
completion probabilities, however, is to serve the students better 
When counseling is made available as early in the course as possible 
we wouid hope that the sense of anonymity inherent in mass education 
techniques can be counterbalanced with human concern for the individual 
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